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XO . SUMMARY 


As detailed in the Statement of Work for this research effort, the principal 
investigator was to: 

(1) survey the experimental efforts and facilities within the NASA/MSFC 
Propulsion Laboratory, 

(2) examine the uncertainty methodologies, approaches, and techniques 
currently used within the laboratory and those currently required of con- 
tractors in propulsion research, 

(3) assess strengths and weaknesses of current uncertainty approaches 
and provide recommendations as appropriate, and 

(4) provide specific recommendations for improvements in error analysis 
and the management of experimental uncertainty to aid in (A) the validation 
of power balance and internal flow codes based on data from the Technology 
Test Bed (TTB) engine and (B) the assessment of the rated performance of 
flight Space Shuttle Main Engines (SSME’s) based on acceptance test and 
flight data analysis. 

During a meeting on August 23, 1991 with Jerry Redus, David Seymour, 
Margie Zoladz and Charles Martin of the Propulsion Laboratory, it was decided 
that the areas of primary initial interest were the uncertainties associated 
with specific impulse determination in SSME ground testing (including the 
TTB program) and in Solid Rocket Motor (SRM) ground testing. Also of 
interest were the uncertainties associated with scale-up of the Space 
Transportation Main Engine (STME) to the final configuration and uncertainty 



considerations when comparing SSME specific impulse values from ground 
tests and from flight data. 

After surveying documents on SSME design and operation and 
beginning to examine the voluminous materials from SSME Performance 
Review Meetings from July 1990 forward, the PI decided to concentrate the 
remainder of this one man-month effort primarily on assessing the uncer- 
tainties associated with SSME Acceptance Tests at Stennis Space Center 
(SSC). The results of this assessment are described in Section 2 of this report. 

Specific conclusions and recommendations are presented in Section 3. 
These can be summarized by stating that it is necessary to consider separately 
the random (precision) and fixed (bias) components of uncertainty when 
evaluating experimental results and comparing results from different tests. 
It is also necessary to consider possible correlated bias error effects, particular- 
ly in the TTB program when results from different tests using the same test 
stand and base engine are compared. 
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2.0 SSME GROUND TEST UNCERTAINTY ASSESSMENT 

, , o 

2.1 Uncertainty Analysis Background 

An experiment is performed to answer a question or to find the solution 
to a problem. Designate the true answer to the question or true solution to the 
problem as r tnie . The result which is obtained from the experiment is r, a 
flawed estimate of r true . To properly interpret the meaning of the result r, an 
estimate of ±U X , the interval around r in which we are 95% confident that r true 
lies. 

The terms necessary to determine the 95% confidence interval covering 
the true result are defined below: 

Precision Limit. P r The ±P r interval about a result is the band within 
which the mean result, p, would fall 95% of the time if the experiment 
were repeated many times under the same conditions using the same 
equipment. The precision limit is a result of the scatter (or lack of 
repeatability) caused by random errors and unsteadiness. 

Bias Limit. B r . The bias limit is an estimate of the magnitude of the 
fixed, constant errors. When the true bias error in a result is defined 
as p, the quantity B r is the experimenter’s 95% confidence estimate such 
that 

IPI*B r 
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Uncertainty. U r The ±U r interval about the result is the band within 
which the experimenter is 95% confident the true value of the result 
lies. The 95% confidence uncertainty is calculated from [1] as 

U r = [B 2 + Pf] ip - (1) 


If an experiment has been repeated a number of times so that M 
previous results are available (such as multiple SSME tests on the same test 
stand), then the best estimate of the precision limit to associate with another 
similar result would be P r calculated as 

P r = tS r (2) 


where t is the 95% confidence level value of the Student’s t distribution for 
v = M - 1 degrees of freedom. S r is the precision index (sample standard 

deviation) of the set of M results and is defined by 




M- 


'rr £ <r. - ^ 


1/2 


(3) 


where r is the mean of the M r k ’s. 

The bias limit, B r , is the experimenter’s 95% confidence estimate of the 
magnitude of the fixed error in the result. When the result r is determined 
from measured values of J variables 

r = r(X : , Xy ... Xj) (4) 
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then the bias limit of the result is related to the bias limits B i of the measure- 
ments of the separate variables Xj fcy 


B 2 r - £ 




M 


(5) 


where 


e,= 


dr 

dX, 


( 6 ) 


p ik is the correlation coefficient for the biases in the measurements of Xj and 
X k , and 8 ik is the Kronecker delta. The bias limits Bj are estimates at 95% 
confidence of the magnitude of the fixed errors in the measurements of the 
separate variables Xj. 

In practice the correlated biases are usually handled by making the 
approximation 

- (!)«,'»* , (7) 


so that 


tf = £ 

M 


w * Ewwi-y 

M 


(8) 
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where B[ and are the portions of biases in measurements of variables 

and that arise from the same sources and are presumed to be perfectly 
correlated (Coleman and Steele, [2]). 

Correlated bias errors are those that are not independent of each other. 
There has been very little discussion in the engineering literature of these 
concepts or their application. The ANSI/ASME Standard on Measurement 
Uncertainty [1] mentions correlated bias errors only in one of the examples 
presented. Coleman and Steele presented a derivation of the propagation 
equation for bias errors including the effects of correlated elemental bias 
sources [3] and also discussions of the approximation of such terms in practical 
applications [2]. 

2.2 SSME Specific Impulse Uncertainty Analysis 

♦ 

The SSME vacuum specific impulse I is determined from 

/ = F m / W a (9) 

IS 9 <L 

where F a is the adjusted thrust and W a is the adjusted mass flow rate of the 
propellants. For tests using SSC test stand A-2, these are determined from 
F a ~ Fi c -F a +F^ + F 0 + Ff + F rb + F UM y+F d y+F cla M + FbaM 

and 

K - (11) 
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The specific impulse is thus a function of 16 variables. 

F| c is the sum of the , forces, measured by the 3 load cells, and it is 
considered as one variable in the uncertainty analysis because the calibration 
procedure effectively calibrates the sum of the outputs of the 3 load cells rather 
than treating them individually. F zg is the sum of the zero shifts measured 
using the 3 load cells prior to engine startup. The next four variables in 
Equation (10) - F pw , F 0 , F f , and F rb -- are corrections for propellant weight, 
oxygen and fuel inlet momentum gains, and for the reaction beam effects, 
respectively. The final four variables in Equation (10) are corrections to 
vacuum conditions, with F undif , F dif , F clam , and F base being adjustments for the 
undiffused exit area, the diffused exit area, the clamshell seal area, and the 
atmospheric base pressure, respectively. 

In Equation (11), p 0 and p f are the oxygen and fuel densities at the 
volumetric flowmeters which measure Q 0 and Q f , and W orp and W frp are the 
oxygen and fuel repressurization mass flow rates. 

The bias limit of the experimental result, I, and the precision limit of the 
experimental result should be considered separately. 

2.2.1 Specific impulse bias limit estimation 

So that specific numerical magnitudes can be investigated, SSME 
Ground Test A2-542 was chosen as a typical "nominal" test, and the results 
from that test are given in Figure 1. 
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FIGURE 1: SSME SPECIFIC IMPULSE 
DETERMINATION FOR TEST A2-542 
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In considering the result from a single test, there are no apparent 
correlated biases of significance. Application of the bias limit propagation 
expression (Equation (8)) to Equations (9), (10), and (11) and algebraically 
manipulating the resulting expression into its simplest form yields the 
equation shown in Figure 2. Also shown in Figure 2 are the values of each of 
the individual bias limit terms assuming the bias limit for each of the 16 
variables is 1%. While we know that this certainly is not so, this view allows 
us to see which variables are most important from an uncertainty standpoint 
when all uncertainties are equal. It is apparent that, under these assump- 
tions, the bias errors in the load cell measurements, in the oxygen volumetric 
flowrate measurement, and in the oxygen density value are the most dominant, 
followed by the bias errors in the fuel volumetric flowrate and density and the 
corrections for the undiflused and diffused exit areas. These results identified 
the variables which should be concentrated upon in this research effort. 

A similar presentation is shown in Figure 3, except that in this case 
estimates for the bias limits identified as being of greatest significance were 
more carefully made. The bias limits shown in this figure are thought to be 
good "ballpark" estimates and result in a value of Bj of 0.33% (or 1.5 seconds). 
The bias limit for the thrust measured by the load cells, B Flc , was estimated 
using the NIST stated accuracy of the calibration standard (=0.04% of the full 
scale of 500,000 lbf) and discussions of the calibration procedure with 
Rocketdyne personnel at SSC. This author thinks that the 0.08% value in 
Figure 3 is the lowest estimate that can be justified and is probably on the low 
side. 
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FIGURE 2: SPECIFIC IMPULSE BIAS LIMIT FOR 
TEST A2-542 IF ALL B’s 1% (B, = 1.5% or 7 sec) 
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FIGURE 3: SPECIFIC IMPULSE BIAS LIMIT FOR 
TEST A2-542 WITH BALLPARK B’s (B, = 0.33% or 1.5 sec) 
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The estimates of 2% for the bias limits on the correction models for the 
undiffused area and diffused .area may be on the low side by as much as a 
factor of 5 or so -- this analysis shows that these uncertainty contributors are 
of potential importance and should be studied further. 

The terms containing the bias limits in the fuel volumetric flowrate and 
the fuel density are not significant contributors, and the estimates of 0.25% are 
thought to be roughly correct. The terms containing the bias limits in the 
oxygen volumetric flowrate and the oxygen density are the largest contributors 
using the assumptions in Figure 3. The 0.25% estimate for Bq 0 is thought to 
be a good "ballpark" value. The 0.25% estimate for the bias limit for oxygen 
density is a good, defensible value; has potentially significant repercussions; 
and thus deserves further discussion. 

This estimate of the bias limit for oxygen density is related to the degree 
of agreement between the original data sets and the curvefit(s) of those data 
from which the density tables were constructed as functions of temperature 
and pressure - it does not consider the effects of bias errors in the measure- 
ments of oxygen temperature and pressure at the test stand flowmeter. 
Examination of an article in the Journal of Research of the NBS (Reference 4), 

f 

which discusses agreement of the curvefits with NBS data and other previously 
reported data, shows that bias limit estimates of 0.25% up to 0.5% are 
reasonable for densities of compressed liquid and saturated liquid oxygen. 
Using this estimate range of 0.25% - 0.50% for the oxygen density bias limit 
and assuming that the bias errors are zero in every one of the other 15 
variables, a range for specific impulse bias limit is calculated as 0.21% - 0.43% 
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or about 1 to 2 seconds. It therefore appears that the fixed errors in the 
oxygen density tables impose a minimum bias limit of 1 to 2 seconds for 
specific impulse determinations, and this certainly must be considered when 
deciding whether measurement systems for the other variables need improve- 
ment. 

2.2.2 Specific impulse precision limit estimation 

Since multiple SSME ground tests have been run, the existing data base 
of specific impulse determinations can be used with Equations (2) and (3) to 
obtain estimates of the precision limit(s). An unpublished analysis presented 
by Rocketdyne personnel at an SSME Performance Review in July 1990 
presented the following SSME Phase II statistics: 

“ 0.46 sec » 0.5 sec 
= 0.86 sec « 1 sec 

If we use t = 2, the precision limits are 

a ^ sec 

(fymg iiw-mgin* 85 2 sec 

2.2.3 Specific impulse overall uncertainty estimation 

Combining the specific impulse bias limit estimates and precision limit 
estimates as in Equation (1) we obtain a range of overall uncertainty as 

Uj ® 1.5 sec — • 3 sec 
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3.0 CONCLUSIONS AND RECOMMENDATIONS 


In planning and designing tests and in interpreting the results of tests, 
the bias and precision components of experimental uncertainty should be 
considered separately. While consideration of only the overall uncertainty is 
better than not considering the effects of experimental errors at all, it does not 
allow one to make the most effective application of resources in experimental 
programs or to fully interpret the implications of the results. 

The effects of correlated bias errors should be considered when 
comparing results from tests which have the same or some of the same error 
sources. Such cases occur, for example, in comparing tests on the same engine 
at different conditions, on the "same" engine before and after modifications, 
and on different engines tested on the same stand. Assuming that "the effects 
of all of the fixed errors subtract out" when interpreting test comparisons is, 
in general, incorrect. 

The magnitude of the precision errors -- the "scatter" in results test-to- 
test and/or engine-to-engine - should be considered when planning a test 
program to determine the effects of design changes. When the anticipated 
effect of the design change is of the same order as the precision error effects, 
further consideration of the test plan or test design is indicated before 
committing resources to the execution of the test. 

The initial uncertainty analysis of the specific impulse determination in 
SSME tests on Stand A-2 at SSC yielded some interesting insights. The likely 
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magnitude of the fixed errors in the oxygen density tables sets a minimum bias 
limit for the specific impulse . result- in the ±1 to ±2 seconds range. More 
detailed analyses of the uncertainties associated with the load cells, the oxygen 
volumetric flowmeter, and the undiffused/diffused area model for correction to 
vacuum conditions are indicated since these have been identified as possible 
significant sources of uncertainty. The statistics of the SSME Phase II ground 
test data indicate a test-to-test precision limit for specific impulse of about ±1 
second and an engine-to-engine precision limit of about ±2 seconds. This 
means that one would expect about 95% of the SSME specific impulse 
determinations on SSC Stand A-2 to fall within a band about 4 seconds wide 
if the engines are the "same" as those previously tested. 

It is recommended that a more detailed analysis of the uncertainties 
mentioned in the previous paragraph be made for the SSME ground tests and 
that an uncertainty analysis be performed on the determination of specific 
impulse from Shuttle flight data so that ground test and flight results can be 
properly compared. The uncertainties associated with the TTB experimental 
program should be estimated, with particular attention to the effects of 
correlated bias errors, the minimum test-to-test scatter that can be expected, 
and the influence of experimental uncertainties on test planning with Taguchi 
methods. In addition, uncertainty analysis techniques should be used to 
analyze SRM ground testing and estimates made of the uncertainties in the 
specific impulse results from those tests. 
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